Evaluating Rater Accuracy in Rater-Mediated Assessments Using an Unfolding Model
نویسندگان
چکیده
منابع مشابه
Approximate measurement invariance in cross-classified rater-mediated assessments
An important assumption underlying meaningful comparisons of scores in rater-mediated assessments is that measurement is commensurate across raters. When raters differentially apply the standards established by an instrument, scores from different raters are on fundamentally different scales and no longer preserve a common meaning and basis for comparison. In this study, we developed a method t...
متن کاملEvaluating frame-of-reference rater training effectiveness using performance schema accuracy.
Frame-of-reference training has been shown to be an effective intervention for improving the accuracy of performance ratings (e.g., Woehr & Huffcutt, 1994). Despite evidence in support of the effectiveness of frame-of-reference training, few studies have empirically addressed the ultimate goal of such training, which is to teach raters to share a common conceptualization of performance (Athey &...
متن کاملOn Rater Agreement and Rater Training
This paper first analyzed two studies on rater factors and rating criteria to raise the problem of rater agreement. After that the author reveals the causes of discrepencies in rating administration by discussing rater variability and rater bias. The author argues that rater bias can not be eliminated completely, we can only reduce the error to a cetain degree by training raters. The study on r...
متن کاملUsing paired comparison matrices to estimate parameters of the partial credit Rasch measurement model for rater-mediated assessments.
The purpose of this paper is to describe a technique for estimating the parameters of a Rasch model that accommodates ordered categories and rater severity. The technique builds on the conditional pairwise algorithm described by Choppin (1968, 1985) and represents an extension of a conditional algorithm described by Garner and Engelhard (2000, 2002) in which parameters appear as the eigenvector...
متن کاملA Hierarchical Rater Model for Constructed Responses, with a Signal Detection Rater Model
The hierarchical rater model (HRM) re-cognizes the hierarchical structure of data that arises when raters score constructed response items. In this approach, raters’ scores are not viewed as being direct indicators of examinee proficiency but rather as indicators of essay quality; the (latent categorical) quality of an examinee’s essay in turn serves as an indicator of the examinee’s proficienc...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
ژورنال
عنوان ژورنال: Educational and Psychological Measurement
سال: 2016
ISSN: 0013-1644,1552-3888
DOI: 10.1177/0013164415621606